Papers with data cleaning methods
The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Open-source dataset of code-text pairs for training large language models to understand code is outperforms other datasets for code generation and understanding tasks. |
| Approach: | They propose to extract high-quality code-text pairs from a dataset of 43 million pairs . they use rules and deep learning to ensure that the code-sampled samples contain high-quality pairs a . |
| Outcome: | The Vault dataset outperforms existing models on common coding tasks . authors hope the results will propel AI research and software development forward . |
How Much Do Robots Understand Rudeness? Challenges in Human-Robot Interaction (2024.lrec-main)
Copied to clipboard
| Challenge: | This paper examines the pressing need to understand and manage inappropriate language within the evolving human-robot interaction landscape. |
| Approach: | They propose to use data cleaning methods to identify inappropriate language in real-time interactions and evaluate natural language models for their proficiency in discerning rudeness. |
| Outcome: | The proposed methods identify and mitigate inappropriate language in real-time interactions and evaluate natural language models for their proficiency in discerning rudeness. |